Judging health systems: reflections on WHO's methods.

The " attainment " indicator values in WHO's World Health Report 2000, whatever their conceptual merits, are spurious: only 39 percent are based on actual country-level observations of appropriate variables—only 18.5 percent, for indicators other than disability-adjusted life expectancy. 61 percent are imputations from regressions based on real data for a few countries. Furthermore, the " responsiveness " indicators are not comparable across countries; and in three cases the values obtained from expert informants were discarded in favour of imputed values, without persuasive justification. The indexes of composite attainment and " performance " for ranking countries are also meaningless, because they are based on these imputations. WHO member governments were not informed of the methods and sometimes suffered unjust criticism over the published rankings. If judgements about performance are worth pursuing, they should be based on real data, represent methodological consensus among governments, be built up from assessments at much less aggregated levels, and be useful for public policy. Readers of the World Health Organisation's World Health Report 2000 [1] will have noticed that although many numbers are presented in the ten Annex tables, there is scant reference in the text to the indicators used to describe " attainments " or " performance " , or their implications for health systems. One critic, remarking on this, says that " Fortunately,…the report appears to make very little connection between the results of the performance analysis and the implications for undertaking [the] functions [of health systems]. " [2]. Part of the reason is that the text authors were told essentially nothing about how some of the indicators were estimated until near the end of production of the Report, and the Annex was sent to the printer some time after the text. Two exceptions to this separation should be noted. First, I participated in many of the discussions concerning the " fair financing " indicator and co-authored two WHO Discussion Papers describing its logic and construction [3-4]. Second, Chapter 2 of the Report, which I wrote, offers brief explanations of all the different indicators, with somewhat more detail being provided in the Explanatory Notes to the Annex. Chapter 2 was the last to be written, and reflects what I learned from several of the team working on the Annex, who are listed under " Working Groups " on p. 206 of the Report, in the last weeks of text production. What follows concerns …